Dynamic backup workers for parallel machine learning

The most popular framework for distributed training of machine learning models is the (synchronous) parameter server (PS). This paradigm consists n workers, which iteratively compute updates model parameters, and a stateful PS, waits aggregates all to generate new estimate parameters sends it back workers iteration. Transient computation slowdowns or transmission delays can intolerably lengthen time each An efficient way mitigate this problem let PS wait only fastest n−b updates, before generating parameters. slowest b are called backup workers. correct choice number depends on cluster configuration workload, but also (as we show in paper) hyper-parameters algorithm current stage training. We propose DBW, an that dynamically decides during process maximize convergence speed at Our experiments DBW (1) removes necessity tune by preliminary time-consuming experiments, (2) makes up factor 3 faster than optimal static configuration.